Skip to content

Conversation

@ChrisRackauckas-Claude
Copy link

Summary

Fixes CI failures in NonlinearSolveSciPy tests on Julia v1.12 caused by Python 3.14.0's buggy stack overflow detection mechanism.

Problem

The CI is failing with:

Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable stack overflow (used -1133875 kB)

Note the negative memory values - this indicates a calculation bug in Python 3.14's new stack overflow detection, not an actual stack overflow.

Root Cause

Python 3.14 introduced a redesigned stack overflow detection mechanism that has multiple known bugs:

  • Produces false positives with negative memory calculations
  • Incompatible with contexts where stack base addresses change (coroutines, Julia tasks, C++ Boost contexts)
  • Affects Julia 1.9, 1.12 on both macOS and Linux

This is not a PythonCall.jl or NonlinearSolveSciPy issue - it's an upstream Python 3.14 bug affecting many projects.

Solution

Added CondaPkg.toml to pin Python to >=3.9,<3.14, ensuring Python 3.13.x is installed instead.

References

Test Plan

CI should pass with Python 3.13.x instead of 3.14.0.

🤖 Generated with Claude Code

ChrisRackauckas and others added 18 commits November 13, 2025 10:58
Fixes CI failures caused by Python 3.14.0's buggy stack overflow detection.
The error "Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable stack
overflow" with negative memory values indicates a calculation bug in Python
3.14's new stack detection mechanism when interacting with Julia's task system.

This issue affects:
- Julia 1.9, 1.12
- macOS and Linux
- Python 3.14.0

Pinning Python to >= 3.9, < 3.14 resolves the issue.

References:
- PythonCall.jl issue: JuliaPy/PythonCall.jl#694
- Python upstream issues: python/cpython#139653, python/cpython#137573

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Fixes test failures where Julia-specific keyword arguments (alias, verbose)
were being forwarded to scipy.optimize functions that don't recognize them.

The error was:
```
Python: TypeError: least_squares() got an unexpected keyword argument 'alias'
Python: TypeError: root() got an unexpected keyword argument 'alias'
```

Now filters out :alias and :verbose kwargs in all three __solve methods:
- SciPyLeastSquares
- SciPyRoot
- SciPyRootScalar

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Fixes two critical issues in Python interop:

1. **Type conversion errors**: Replace direct Vector{Float64}() and Float64()
   constructors with pyconvert() for proper PythonCall type conversion.
   - Import pyconvert from PythonCall
   - Update _make_py_residual to use pyconvert(Vector{Float64}, x_py)
   - Update _make_py_scalar to use pyconvert(Float64, x_py)
   - Update all result conversions (res.x, res.fun) to use pyconvert

   Fixes error:
   ```
   MethodError: no method matching Vector{Float64}(::PythonCall.Py)
   ```

2. **Kwargs splatting errors**: Change from filter() iterator to pairs generator
   for proper kwargs splatting to Python functions.
   - Replace scipy_kwargs = filter(...) with inline pairs generator
   - Use (k => v for (k, v) in pairs(kwargs) if k ∉ (:alias, :verbose))...

   Fixes error:
   ```
   TypeError: object of type 'NoneType' has no len()
   ```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Fixes two critical issues discovered in CI:

1. **Kwargs splatting error**: Convert generator to Tuple before splatting
   - Change from inline generator: `(k => v for ...)...`
   - To collected Tuple: `scipy_kwargs = Tuple(k => v for ...); scipy_kwargs...`
   - Fixes: `TypeError: object of type 'NoneType' has no len()`

2. **Boolean context errors**: Use pyconvert for Python boolean fields
   - `res.success` → `pyconvert(Bool, res.success)`
   - `res.converged` → `pyconvert(Bool, res.converged)`
   - Fixes: `TypeError: non-boolean (PythonCall.Py) used in boolean context`

Applied to all three solver methods:
- SciPyLeastSquares (line 137, 151)
- SciPyRoot (line 177, 190)
- SciPyRootScalar (line 218, 230)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Fixes error: `TypeError: object of type 'NoneType' has no len()`

Problem: We were passing `bounds=None` to scipy when bounds weren't specified,
but scipy.optimize.least_squares tries to call len() on the bounds parameter,
which fails on None.

Solution: Conditionally omit the bounds kwarg entirely when not specified,
rather than passing Python's None. This allows scipy to use its default
bounds of (-inf, inf).

Split the scipy call into two branches:
- Without bounds: When prob doesn't have lb/ub constraints
- With bounds: When prob has lb/ub constraints

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
IntervalNonlinearProblem doesn't have a u0 field (it uses tspan for the
bracket). Added conditional check using hasfield() before accessing prob.u0
in both solve() and init() functions to handle problem types that don't
have initial conditions.

Fixes error: "type IntervalNonlinearProblem has no field u0" when solving
with SciPyRootScalar.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
IntervalNonlinearProblem doesn't have a u0 field, so it should bypass the
NonlinearSolveBase solve machinery by directly overloading CommonSolve.solve
instead of SciMLBase.__solve. This avoids the need for conditional checks
in the base solve function.

- Changed SciPyRootScalar to overload CommonSolve.solve for IntervalNonlinearProblem
- Added using CommonSolve to NonlinearSolveSciPy
- Reverted the hasfield() conditional checks in NonlinearSolveBase since they're no longer needed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Added CommonSolve to dependencies since we use CommonSolve.solve for
IntervalNonlinearProblem.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Convert all Python statistics fields (nfev, njev, nit, function_calls,
iterations) to Julia Int using pyconvert instead of direct Int() constructor.
Also convert u_root from Python Float to Julia Float64 in SciPyRootScalar.

Fixes:
- MethodError: Cannot convert PythonCall.Py to Int64 in NLStats
- MethodError: no method matching ndims(::PythonCall.Py) in build_solution

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Added :lb and :ub to the kwargs filter list to prevent them from being
forwarded to NonlinearSolveBase which doesn't recognize them as valid
solve kwargs. These bounds are extracted directly from the problem
using hasproperty and passed to scipy separately.

Fixes: Unrecognized keyword arguments error for [:lb, :ub]

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Created a custom keyword argument handler NonlinearKeywordArgError that
extends the standard SciMLBase keywords to include bounds (lb, ub) for
NonlinearLeastSquaresProblem.

- Added NonlinearKeywordArgError struct and checkkwargs method in NonlinearSolveBase.jl
- Added specific solve_call and init_call methods for NonlinearLeastSquaresProblem
  that use NonlinearKeywordArgError as the default kwargshandle
- This allows bounds to be passed as problem kwargs without triggering
  "Unrecognized keyword arguments" errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Filter out :lb and :ub from kwargs passed to scipy since these bounds
are extracted directly from the problem using hasproperty and passed to
scipy separately via the bounds parameter.

This is a minimal, non-invasive fix that only touches NonlinearSolveSciPy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
…blem

Updated NonlinearSolveSciPy and NonlinearSolveBase to use the new lb/ub
fields added to NonlinearLeastSquaresProblem in SciMLBase.jl PR #1169:

NonlinearSolveSciPy:
- Import and use SciMLBase.allowsbounds trait
- Implement allowsbounds trait methods:
  - allowsbounds(::SciPyLeastSquares) = true
  - allowsbounds(::SciPyRoot) = false
  - allowsbounds(::SciPyRootScalar) = false
- Update __solve to get bounds from prob.lb and prob.ub instead of hasproperty

NonlinearSolveBase:
- Add bounds checking in solve_call for NonlinearLeastSquaresProblem
- Error if algorithm doesn't support bounds but problem has them

This properly separates concerns:
- SciMLBase defines the problem interface with lb/ub fields
- allowsbounds trait indicates algorithm support
- NonlinearSolveBase validates compatibility
- NonlinearSolveSciPy uses the fields directly

Depends on: SciMLBase.jl #1169

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Extended bounds checking in NonlinearSolveBase to support both
NonlinearProblem and NonlinearLeastSquaresProblem with lb/ub fields.

Updated SciMLBase PR to add lb/ub fields to both problem types:
- Removed unnecessary UNSET_BOUNDS constant
- Added lb/ub fields to NonlinearProblem
- Updated ConstructionBase.constructorof for both problem types

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Updated SciMLBase compat bounds to require version 2.127 which includes
the lb and ub fields added to NonlinearProblem and NonlinearLeastSquaresProblem.

Updated in:
- Main Project.toml
- lib/NonlinearSolveBase/Project.toml
- lib/NonlinearSolveSciPy/Project.toml

Closes dependency on SciMLBase.jl #1169

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@ChrisRackauckas ChrisRackauckas merged commit 894de94 into SciML:master Nov 14, 2025
74 of 98 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants